Pilot-Streaming: A Stream Processing Framework for High-Performance Computing
نویسندگان
چکیده
An increasing number of scientific applications rely on stream processing for generating timely insights from data feeds of scientific instruments, simulations, and Internet-of-Thing (IoT) sensors. The development of streaming applications is a complex task and requires the integration of heterogeneous, distributed infrastructure, frameworks, middleware and application components. Different application components are often written in different languages using different abstractions and frameworks. Often, additional components, such as a message broker (e. g. Kafka), are required to decouple data production and consumptions and avoiding issues, such as back-pressure. Streaming applications may be extremely dynamic due to factors, such as variable data rates caused by the data source, adaptive sampling techniques or network congestions, variable processing loads caused by usage of different machine learning algorithms. As a result application-level resource management that can respond to changes in one of these factors is critical. We propose PilotStreaming, a framework for supporting streaming frameworks, applications and their resource management needs on HPC infrastructure. Pilot-Streaming is based on the Pilot-Job concept and enables developers to manage distributed computing and data resources for complex streaming applications. It enables applications to dynamically respond to resource requirements by adding/removing resources at runtime. This capability is critical for balancing complex streaming pipelines. To address the complexity in developing and characterization of streaming applications, we present the Streaming MiniApp framework, which supports different plug-able algorithms for data generation and processing, e. g., for reconstructing light source images using different techniques. We utilize the Mini-App framework to conduct an evaluation of the Pilot-Streaming capabilities.
منابع مشابه
Pilot-Streaming: Design Considerations for a Stream Processing Framework for High-Performance Computing
Streaming capabilities are becoming increasingly important for scientific applications [1], [2] supporting important needs, such as the ability to act on incoming data and steering. The interoperable use of streaming data sources within HPC environments is a critical capability for an emerging set of applications. Scientific instruments, such as x-ray light sources (e. g., the Advanced Photon S...
متن کاملMapping Streaming Applications to OpenCL
Graphic processing units (GPUs) have been gaining popularity in general purpose and high performance computing. A GPU is made up of a number of streaming multiprocessors (SM), each of which consists of many processing cores. A large number of general-purpose applications have been mapped onto GPUs efficiently. Stream processing applications, however, exhibit properties such as unfavorable data ...
متن کاملLow Latency Stream Processing: Twitter Heron with Infiniband and Omni-Path
Worldwide data production is increasing both in volume and velocity, and with this acceleration, data needs to be processed in streaming settings as opposed to the traditional store and process model. Distributed streaming frameworks are designed to process such data in real time with reasonable time constraints. Twitter Heron is a production ready large scale distributed stream processing fram...
متن کاملReal-Time Stream Processing in Java
This paper presents a streaming data framework for the Real-Time Specification for Java, with the goal of levering as much as possible the Java 8 Stream processing framework whilst delivering bounded latency. Our approach is to buffer the incoming streaming data into micro batches which are then converted to collections for processing by the Java 8 infrastructure which is configured with a real...
متن کاملStream Data Mining: Platforms, Algorithms, Performance Evaluators and Research Trends
Streaming data are potentially infinite sequence of incoming data at very high speed and may evolve over the time. This causes several challenges in mining large scale high speed data streams in real time. Hence, this field has gained a lot of attention of researchers in previous years. This paper discusses various challenges associated with mining such data streams. Several available stream da...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
- CoRR
دوره abs/1801.08648 شماره
صفحات -
تاریخ انتشار 2018